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COMPUTERIZED SYSTEM AND METHOD FOR SOLICITING, 
COLLECTING AND EVALUATING SUGGESTIONS FROM STRUCTURED AND 
UNSTRUCTURED STIMULI VIA A COMPUTER NETWORK 

FIELD OF THE INVENTION 

The present invention relates in general to methods of acquiring useful 
information via a computer network, such as the Internet. More specifically, 
the present invention relates to a method and apparatus enabling individuals 
or institutional entities to obtain and provide real-time, anonymous, and 
subject-specific feedback to improve their information gathering, information 
providing, and decision-making capabilities. In particular, the present invention 
relates to a method and apparatus for soliciting and organizing suggestions 
(or feedback) in electronic format via a computer network, such as the 
Internet, using electronic suggestion boxes; and processing such feedback 
information by evaluation, filter, and knowledge extraction mechanisms 
before delivering to the entities that seek the feedback. The present 
invention further relates to an assembly of individual suggestion boxes, inter- 
connected on the World Wide Web, and thus allows sharing of anonymous 
feedback information among consenting users. 
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BACKGROUND OF THE INVENTION 

Suggestion boxes have been used by "organizations' 7 (i.e., business, 
government and non-profits) to solicit suggestions and feedback from their 
members and constituents. Perhaps the most common type of conventional 
suggestion collection system is the one associated with the purchase of 
goods or services. A consumer can usually provide feedback to a merchant 
by filling out a questionnaire and dropping it off in a physical suggestion box 
at the store, or by sending a letter or completed questionnaire to the 
company via the postal service, or electronically, by clicking on a 
"suggestion" or "feedback" icon/link on the company's Web site. 

It is noteworthy that the use of suggestion boxes associated with 
"private individuals" (e.g., consumers, employees, members of organizations) 
is rare, if at all present, in existing suggestion collection schemes. For 
example, it is difficult for a given individual to gather feedback, in an 
anonymous fashion, from other individual(s) with whom he or she may be 
interacting with for business or other purposes, and whose opinion may be 
of interest or importance to him or her in specific areas. In addition, 
conventional suggestion collection systems are also limited with respect to 
their accessibility, anonymity, privacy, and evaluation. 

Great accessibility is desirable because it will increase the volume of 
the data collected, as well as enhance the diversity, thus increase the 
representativeness of the data. To receive a large and diverse quantity of 
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data, physical suggestion boxes need to be situated in as many locations as 
possible. In the example of individuals, this task is complex and burdensome 
because one may need to interact with a myriad of people in a myriad of 
places. Also, it may be impracticable for an individual to place physical 

5 suggestion boxes over a large geographic area, due to various constraints 
(costly to rent private property for such placement). The problem of 
accessibility is less pronounced for organizations, since they generally 
occupy the real property where they transact with customers or where 
employees congregate to work, and the customers and employees are the 

10 constituents they care to receive suggestions from. However, the lack of 
flexibility in physical suggestion boxes remains a significant obstacle for their 
accessibility. 

Electronic implementations of suggestion boxes allow an individual or 
institutional entity to use a designated site on the Web to gather suggestions 

15 from a potentially large and diversified group of contributors. Typically, web 
sites collect suggestions or feedback information electronically via dedicated 
e-mail addresses (e.g., suggestions@xyzcorp.com) or html forms accessible 
directly on the web site. Additionally, electronic forums (e.g., bulletin boards 
and chat rooms) on the recipient's or a third-party's web site may be used 

20 for the same purpose, where suggestions and feedback are submitted and 
viewed by the general public. The seemingly improved accessibility, 
however, provides little benefit to individual private users because most of 



3 
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them do not have the facility to create and maintain their own web site, and 
to receive significant number of hits required to gain meaningful pools of 
suggestions. Meanwhile, the available web-based suggestion-collection 
systems do not offer mechanisms for organizing, evaluating, and filtering 

5 information gathered. Therefore, a new paradigm of collecting and managing 
suggestion information with consistent high accessibility is needed. 

Anonymity is an important factor for any system that is designed to 
effectively deal with information related to people. The e-mail or bulletin 
board-based suggestion collection systems discussed above, for example, do 

10 not afford desired anonymity, because a contributor's identity is easily 
discoverable though various tracking techniques over the Internet, such as 
using cookies. Current e-mail-based suggestion boxes also inevitably require 
a return e-mail address, which tend to discourage contributors from 
submitting suggestions in some situations when they are concerned about 

15 revealing their identity. The lack of anonymity in these existing suggestion 
collection systems also impairs the web site owners' ability to fashion 
incentives and monetary rewards to effectively encourage suggestion 
submission, since contributors may be reluctant to disclose who they are in 
order to receive the reward. Therefore, better resolutions for the anonymity 

20 issue are desired to liberate potential contributors, hence to secure a stable 
volume and diversified resources for suggestion or information of interest. 
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Inseparable from the anonymity issues are the concern of privacy, For 
example, an electronic forum, one form of the current suggestion system, 
may discourage potential suggestion recipients from seeking certain 
feedback, because the posted data is of a public nature. This leads to 

5 compromised volume and completeness of the collectable information. The 
seeker of a suggestion is more likely to reveal important issues, often 
sensitive and private, in constructing stimuli to obtain feedback, if he or she 
is ensured that the information gathered can only be viewed by him or her 
self in a secure and private manner. Similarly, in certain situations, a 

10 provider of a suggestion is more likely to submit a suggestion on sensitive 
issues when such submission can be carried out in a private manner. 

Evaluation is helpful for a suggestion collection and managing system. 
As data may grow exponentially through efficient stimuli and incentive- 
induced collecting, the task of procedurally and intellectually evaluating and 

15 processing the data becomes increasingly important. The traditional and 
limited evaluation by the data-recipients themselves often results in biased 
conclusions due to their preexisting dispositions. 

The issue of evaluation also implicates the failure of current 
suggestion collection systems to learn from common themes that transcend 

20 various unaffiliated suggestion boxes, since these systems operate 

independently of each other (e.g., through their respective and distinct web 
sites). It is impossible for the existing systems to provide indirect feedback 
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on a topic of broadly-shared interest, for example, to an organization or 
individual, when no direct feedback to this organization or individual has 
been received. The utility of such systems is therefore drastically limited. 

It is clear that aforementioned drawbacks related to accessibility, 
anonymity, privacy, and evaluation render the currently available suggestion 
collection and managing systems less than desirable for the effective 
acquiring and using of feedback information in the new Information Age. An 
improved system with higher accessibility, better anonymity and privacy, and 
more sophisticated evaluation mechanisms is needed by both institutional 
and individual entities in their pursuance of useful feedback information. 



SUMMARY OF THE INVENTION 

To resolve the above problems, the present invention is directed to a 
computerized suggestion solicitation, collection and evaluation system and 
method, which uses a computer network, such as the Internet, to provide 
real-time, anonymous feedback to individuals and organizations improve their 
information gathering, information providing, and decision-making capabilities. A 
"suggestion" is defined to be any commentary, opinion, idea, factual 
statement, and the like, in various formats such as text, graphical, audio, 
video, and the like. Suggestions may be obtained in response to a stimulus, 
i.e., a questionnaire, a graphic or video image, an audio recording, and a 
physical encounter. The recipient of a suggestion is referred to as a 

6 
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"suggestionee." The provider of a suggestion is referred to as a 
"suggestioner." 

In accordance with one aspect of the invention, a system and method 
are provided for collecting one or more suggestions from a first entity (a 

5 suggestioner) and providing collected suggestions to a second entity (a 
suggestionee) comprising a memory for storing one or more suggestions 
submitted by the first entity in response to a stimulus selected by the second 
entity and an interface permitting the second entity to access the stored 
suggestions. The suggestions may be released in unedited ("raw") form to 

10 the suggestionee, or they may be formatted and transformed to evaluated 
feedback before being delivered to the suggestionee. 

It is an object of the invention to provide a computerized system that 
displays, controls and facilitates the submission and evaluation of 
anonymous suggestions from a large universe of participants that are inter- 

15 connected via a computer network, such as the Internet. It is a further 

object to provide an integrated information collecting and processing system 
that provides a middle layer between the information provider and the 
information receiver. Passing the suggestions through a middle layer makes 
it possible to optionally provide anonymity to the suggestioner and to 

20 optionally analyze or format the suggestions before they are accessed by the 
suggestionee. 
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In accordance with another aspect of the invention, the system 
achieves high level of accessibility by allowing both institutional and private 
users to license the use of the system for their particularized needs. As 
licensed users or subscribers, people can easily seek, obtain, or submit 
suggestions through the system. 

In accordance with yet another aspect of the invention, the anonymity 
of the persons or organizations seeking or submitting suggestions is 
ensured, because the system operates as a third-party trustee of their 
identity information (e.g., name, e-mail address, Uniform Resource Locator, 
Internet Protocol address, etc.). Similarly, the participants may be 
guaranteed privacy whether they submit suggestions or seek feedback 
information in any particular area, since the present system affords them an 
insulated or personalized scope. The suggestioners and suggestionees can 
also disclose their identity at their own discretion thereby making the system 
anonymous in both directions or one direction based on the users' decision. 

In accordance with another aspect of the invention, the present 
system provides an "evaluator," for evaluating collected information and 
organizing it into meaningful suggestions to be delivered to the 
suggestionee. An evaluator can be an automated process or agent 
preconfigured in the system, or an individual or institutional entity registered 
to participate as an evaluator, rather than a suggestioner or suggestionee. 
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In accordance with another aspect of the invention, the basic building 
block of the present system is a "suggestion box," which is implemented in 
the form of a set of computer programs and related databases. A suggestion 
box is assigned to a specific suggestionee; it is used to gather and process 
suggestions he or she desires. A suggestion box comprises five major sub- 
components: 1) a "Suggestion Stimuli Input Tray"; 2) a "Raw Suggestion 
Output Tray"; 3) an "Evaluated Suggestion Output Tray"; 4) a "Create 
Suggestion-Stimuli Process"; and 5) a "Suggestion Evaluation Process". The 
Suggestion Stimuli Input Tray is openly accessible to the public or a defined 
group of participants according to the suggestionee. The other four sub- 
components are private to the suggestionee; a unique and secure computer 
identification and password scheme is used. 

In accordance with another aspect of the invention, a Network 
Commonality Determination Process identifies the common themes among 
suggestion-box stimuli and responses, and establish sharable suggestions to 
participants having similar interests. Indirect feedback information can 
therefore be provided among consenting participants, which is particularly 
useful when stimuli-induced direct suggestions are lacking. 

In accordance with the another aspect of the invention, a Redeem 
Awards process allows the system to use incentives, such as monetary 
awards, to encourage participation, maintain a steady information flow, 
hence effectively generate and deliver useful suggestions. 



9 



WO 01/75683 PCT/US01/09835 

The present invention, therefore, brings significant improvements to 
existing methods and facilities for obtaining meaningful feedback information 
-in a business setting, academic world, or private persons' everyday life. 



BRIEF DESCRIPTION OF THE DRAWINGS 

These and other features will now be described with reference to the 
drawings summarized below. These drawings and the associated 
descriptions are provided to illustrate preferred embodiments of the 
invention, and not to limit the scope of the invention. Throughout the 
drawings, reference numbers are reused to indicate correspondence between 
referenced elements. In addition, the first digit of each reference number 
indicates the figure in which the element first appears. 

FIG. 1 is a diagrammatic illustration of a computerized system for 
suggestion solicitation, collection, and evaluation, according to an 
embodiment of the present invention; 

FIG. 2 is a diagram illustrating a layered architecture of the 
computerized system according to the present invention; 

FIG. 3 is a diagram illustrating a set of processes or modules in the 
computerized system according to the present invention; 

FIG. 4 is a flowchart illustrating an example of a sequence of steps of 
the Open Account 301 shown in FIG. 3; 
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FIG. 5 is a flowchart illustrating an example of a sequence of steps of 
the Create Suggestion-Stimuli Process 303 shown in FIG, 3; 

FIG. 6 is a flowchart illustrating an example of a sequence of steps of 
the Suggestion-Stimuli Input Tray 305 shown in FIG. 3; 

FIG. 7 is a flowchart illustrating an example of a sequence of steps of 
the Suggestion Evaluation Process 307 shown in FIG. 3; 

FIG. 8 is a flowchart illustrating an example of a sequence of steps of 
the Raw Suggestion Output Tray 309 shown in FIG. 3; 

FIG. 9 is a flowchart illustrating an example of a sequence of steps of 
the Network Commonality Determination Process 31 1 shown in FIG. 3; 

FIG. 10 is a flowchart illustrating an example of a sequence of steps 
of the Evaluated Suggestion Output Tray 313 shown in FIG. 3; 

FIG. 11 is a flowchart illustrating an example of a sequence of steps 
of the Redeem Awards 315 shown in FIG. 3. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Referring to Fig. 1, the computerized suggestion system in a preferred 
embodiment of the invention operates in a client-server environment, where 
users communicate via the Internet 111. Users may be a plurality of persons 
or organizations that either solicit suggestions (suggestionee 103), submit 
suggestions (suggestioner 105), or evaluate suggestions (evaluator 107). 
Each user is a client when he or she connects to the system via a web 

11 
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browser interface (1 1 3 for suggestionees, 1 1 5 for suggestioners, and 117 
for evaluators, respectively). 

The sever engine 101 manages all processes in the computerized 
suggestion system, including producing and dispatching stimuli, gathering 
and evaluating suggestions, and targeted suggestion delivering. Various 
types of information processed by the system are stored in a suitable 
database in one embodiment of the invention, or a set of interconnected 
databases in another embodiment of the invention. For example, dedicated 
suggestionee database 121, suggestioner database 123, and evaluator 
database 125 may be established. Particularly relevant in technical 
applications of the present system, a separate database registering expert 
evaluators for certain knowledge domains (expert database 129) is included. 
A database maintaining rewards information (rewards database 1 27) and 
another database for compiling sharable suggestion information 
(commonality database 131) may also optionally be implemented in this 
embodiment of the invention. 

Referring to Fig. 2, the computerized suggestion system according to 
the present invention contains an intermediate processing layer 205 between 
the information receiver/suggestionee layer 201 and the information 
provider/suggestioner layer 203. Suggestionee produces stimuli for potential 
suggestions, which was integrated in the middle processing layer 205 and 
was dispatched to the suggestioner layer 203. Induced by such stimuli, 
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suggestioners submit suggestions to the system, which are sent to the 
middle processing layer 205 to go through the evaluation process. 
Evaluation strategies correspond to the type of stimuli used, and hence 
ensure that the meaningful and desired suggestions, or advice and ideas, can 
be extracted for delivering to the suggestionee. Multiple suggestioners may 
submit various suggestions to a suggestionee via one or multiple suggestion 
boxes. One suggestionee may host multiple suggestion boxes receiving 
suggestions from a plurality of suggestionees. Between a specific 
suggestioner and a specific suggestionee, multiple suggestion boxes 
configured with different purposes and functions may be used for 
information exchange. 

The mid processing layer 205 may be configured to perform a 
plurality of evaluation steps according to the present invention. An individual 
or business entity registered with the system as an evaluator can manually 
review, categorize, and summarize the submitted suggestion to produce 
deliverable suggestion "knowledge." An expert evaluator, in addition, 
focuses on evaluating the soundness of a suggestion based on a specific 
technical or business area, and synthesizing from the raw suggestion 
submissions the high-level "knowledge" or "theme," in response to the 
suggestionee's stimuli. Thus, a suggestionee is provided with a spectrum of 
suggestion products by the computerized suggestion system, ranging from 
raw suggestion submissions to processed and integrated suggestion 
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knowledge, according to the particular needs of the suggestionee. In other 
embodiments of the present invention, automated evaluation steps are used 
to perform statistical analysis, text parsing for words of interest, for 
example. Sophisticated knowledge mining techniques such as machine 
learning are used for analyzing complex data submissions. These processes 
and steps are implemented in the form of computer programs, and operate 
under the synchronized control of the present system. 

Fig. 3 illustrates the processes and modules in the computerized 
suggestion system according to the present invention. First, a user opens an 
account with the system and enters the demographic information (301). A 
suggestionee that seeks suggestions then creates customized stimuli (303) 
to induce suggestions and feedback from potential suggestioners, who will 
subsequently be presented with the customized stimuli (305). Suggestions 
gathered by the system are delivered to the suggestionee in a raw form 
(309) or evaluated form (313). Suggestion evaluation is managed by the 
system (307), as discussed above. In addition, the system also allows the 
suggestionee to evaluate raw submissions him or herself using the evaluation 
tools provided in the system; or to specifically designate a group of 
evaluators. Network Commonality Determination Process 31 1 identifies 
potential common themes, i.e., similarity and correlation between various 
stimuli and corresponding responses. This allows a suggestionee to access 
indirect yet relevant suggestion information, although there is no direct 
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stimuli-induced submission available. Such mechanism of sharing is based 
on participants' voluntary consent, it increases the applicability of the 
information, as well as enhances the accessibility. The effectiveness of the 
computerized suggestion system according to the present invention is 
supported by an incentive awarding process, Redeem Awards (315). 
Suggestioners or evaluators earn rewards for the submissions and evaluation 
they made. Each of the above processes and modules are described in more 
detain in the following description. 

(1) OPEN ACCOUNT 

The Open Account process registers a user on the system either as a 
Suggestionee (the recipient of suggestions), a Suggestioner (the provider of 
suggestions) or as an Evaluator (one that evaluates previously submitted 
suggestions). During the process of registering, the system captures 
demographic information associated with the user. The true identity (name 
or e-mail address) may be volunteered by the user, but is not required as part 
of the registration process. 

Referring to Fig. 4, the Opening Account process 301 in an 
embodiment of the present invention includes a sequence of steps. The 
system creates a user id and password 401 and records background 
information, such as demographic data 403. The system then prompts the 
user how he or she want to register and processes the input accordingly. If 
registered as a suggestionee, the record is stored in suggsetionee database 

15 
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1 21 ; if registered as a suggestioner, the record is stored in suggestioner 
database 1 23; and if registered as an evaluator, the record is stored in 
evaluator database 125. 

(2) CREATE SUGGESTION-STIMULI PROCESS 

Referring to Fig. 5, the Create Suggestion Stimuli Process 303 
includes a sequence of steps to produce suggestion stimuli, in various 
formats including textual, graphical, multimedia, etc. A suggsetionee 
initiates the process by logging onto the system using the designated user id 
and password. The choices of generating structural or non-structural 
questionnaires are presented. Experts' input are needed to control the 
format of structural questionnaires (501), whereas unstructured 
questionnaires take a much more relaxed and flexible form (503). To 
produce structured questionnaires, the system conducts a computerized 
interview with the suggestionee. A series of questions are presented to the 
suggestionee, which are pre-compiled in the system based on the expert 
knowledge in the suggestionee's field, e.g., from a psychologist, a 
motivational coach, or a management consultant. Such expert's knowledge 
on specific matters is previously categorized and stored in a linked computer 
database, the expert database 129. The response from the suggestionee is 
used to either retrieve a list of previously created questionnaires from which 
the suggestionee can choose, or to formulate a new questionnaire. The 
purpose is to create a best-suited document for eliciting suggestions that 

16 
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would effectively assist the suggestionee in achieving certain specific 
objectives. 

In one embodiment of the present invention, the examples of 
questionnaires include the ones related to health, parenting or financial goals; 
musical performances, meetings or presentations; business reports or 
correspondence; and comments on efficiency improvement, working 
conditions, new product, compensation, etc. 

The questionnaires (or other similar stimuli) are placed in the 
Suggestion Stimuli Input Tray 305 to allow simultaneously cross-reference in 
a separate computer database, the commonality database 131 in subsequent 
analyses, as described below (see Network Commonality Determination 
Process). Briefly, the suggestionee is presented with the option of allowing 
others to be able to view this questionnaire, and any associated responses, 
in an anonymous fashion. This feature makes indirect suggestions available 
to other participating suggestionees, i.e., they may see responses to similar 
questionnaires although no direct suggestions to their own and similar 
questionnaire are received). Setting the Commonality Flag to "TRUE 7 ' (505) 
enables this feature for a given questionnaire; and a setting of "FALSE" 
disables this feature. This setting is maintained in the suggestionee's record 
within the Suggestionee Database .121. 

Furthermore, the Create Suggestion-Stimuli Process 303 allows a 
suggestionee to establish incentive awards (e.g., monies, goods or services) 
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to be subsequently displayed along with the suggestion stimuli in the 
Suggestion Stimuli Input Tray 305. This mechanism is to promote and 
encourage the submission of thoughtful anonymous suggestions; it works in 
the following way. If a suggestionee chooses to offer incentives for a 
particular stimuli, the system interacts with the suggestionee to determine 
the amount, type, and conditions of the rewards (507) and sets up an 
escrow account (509) to deposit any reward monies. The Rewards 
Database 127 is subsequently updated (511). 

In one embodiment of the present invention, the final step of the 
Create Suggestion Stimuli Process 303 gives a suggestionee the option of 
creating an e-mail distribution list to inform potential suggestioners of the 
newly created contents in his Suggestion Stimuli Input Tray 305. If choose 
to use this option, the suggestionee is prompted to provide an e-mail 
distribution list (513); this list is subsequently entered into the suggestionee 
database 121 . 

(3) SUGGESTION STIMULI INPUT TRAY 

The goal of the Suggestion Stimuli Input Tray 305 is to provoke 
structured and/or unstructured responses (i.e., suggestions) to subject 
matter that is of interest to a suggestionee. The stimuli in the Suggestion 
Stimuli Input Tray 305 include questionnaires, and any other textual, visual, 
or audio and video presentations, that have been created by the Create 
Suggestion Stimuli Process 303. In addition, the Suggestion Stimuli Input 
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Tray 305 is used to display an incentive award (e.g., a given amount of 
money, a piece of consumer goods, or the enjoyment of certain type of 
services) to promote and encourage the submission of suggestions. 

Referring to Fig. 6, the Suggestion Stimuli Input Tray 305 involves a 
series of operations in one embodiment of the present invention. Once a 
suggestioner logs onto the host web site of the computerized suggestion 
system, he or she is prompted to enter a search term to retrieve an specific 
suggestion box associated with a registered suggestionee (601) that is of 
interest. The suggestioner may choose the intended suggestionee, and 
download the contents of the suggestionee's Suggestion Stimuli Input Tray 
305 to his own computer and view the contents (603) through the web 
browser 1 1 5. The Suggestioner may submit suggestions in response to any 
of the listed suggestion stimuli (e.g., various questionnaires). If the intended 
suggestionee is not found to have been registered on the system, the 
suggestioner may create a temporary suggestion box (608) and submit a 
private, anonymous suggestion to the unregistered suggestionee. The 
system prompts the suggestioner for the e-mail address (or other contact 
information, such as a postal address or telephone number) of the 
suggestionee (609), creates a user id and password (610) and informs the 
suggestionee that an unsolicited suggestion has been received by the 
system. The email notification contains the user id and password, which 
may be used to unlock the private, temporary suggestion box and allow the 
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suggestionee to view the content via the Raw Suggestion Output Tray (See 
Fig. 8). The temporary suggestion (content) and the user id and password of 
the unregistered suggestionee is subsequently stored in the Suggestionee 
Database (610). 

In an embodiment of the invention, the suggestioner is not required to 
open an account on the system, in which case he or she may not engage in 
anonymous bi-directional communication nor share any available rewards 
offered in the system. If, instead, the suggetioner chooses to open an 
account on the system, the Open Account Process 301 is executed. In 
another embodiment of the invention, this allows a suggestioner to uses a 
unique id and password when submitting a suggestion, and each such 
submission to be identified by a unique suggestion number. An option of 
enabling future bi-directional communications with the suggestionee (605) is 
then presented to the suggestioner, which allows the suggestionee to seek 
further clarification and feedback from the suggestioner. Furthermore, if the 
suggestionee has established a reward for the stimuli to which the 
suggestioner responded with a suggestion, the suggestioner is given the 
reward based on established criteria (607). 

(4) RAW SUGGESTION OUTPUT TRAY 

The Raw Suggestion Output Tray 309 presents suggestions gathered 
from suggestioners in their original form. A suggestionee may view the 
contents of his Raw Suggestion Output Tray 309 once he enters the unique 
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user identification number and password. Further, a suggestionee may 
communicate with an anonymous suggestioner if this privilege has been 
given by the suggestionee at the time when the suggestion was submitted. 
This bi-directional communication allows the suggestionee to seek 
clarification or additional data in a secure and anonymous manner from the 
suggestioner on a specific suggestion submission. 

Referring to Fig. 8, the Raw Suggestion Output Tray 309 involves a 
sequence of operations. At the host web site of the system, a suggestionee 
enters his unique user identification and password (801). The suggestionee 
then selects a specific stimuli (803), e.g., a specific questionnaire, from a list 
of presented stimuli that have been previously created in the Create 
Suggestion-Stimuli Process 303. The system accordingly presents a list of 
suggestions gathered in response to that stimulus, which the suggestionee 
can browse through (805). The suggestionee may select an individual 
response and request that additional information/clarification be provided by 
the suggestioner (807). This is accomplished by querying the suggestioner 
database 1 23 to determine if the necessary bi-directional communication has 
been granted by the suggestioner. If such communication was enabled, the 
suggestionee can enter a clarification request (807) that is then forwarded to 
the suggestioner via e-mail (809). 

(5) SUGGESTION EVALUATION PROCESS 
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The Suggestion Evaluation Process 307 allows suggestions to be 
evaluated and organized before delivered to a suggestionee, by a group of 
evaluators, experts, or automated system processes. In one embodiment of 
the invention, a suggestionee identifies the set of suggestions to be 
evaluated; and the system automatically recommends suitable evaluators, 
e.g., a group of computer scientists, for the suggestions, based on 
demographic or other type of information related to various evaluators. In 
another embodiment, the suggestionee may set up a new group of 
evaluators; and recommend each evaluator in the group by entering their 
identity information to the system. Each evaluator is assigned a random 
subset of the suggestions collected in response to a stimuli to process. 

Referring to Fig. 7, the Suggestion Evaluation Process 307 involves a 
sequence of detailed operations in one embodiment of the invention. A 
suggestionee first enters his unique user identification and password, and 
then selects a specific stimulus (701), such as a questionnaire, from a 
presented list of stimuli that have been created in the Create Suggestion- 
Stimuli Process 303. The system displays a list of collected suggestion 
responses to the selected stimulus, along with a list of available evaluators 
or automated evaluation processes that may be suitable for evaluating the 
suggestions (703). The suggestionee may either accept the recommendation 
of evaluators by the system, or create a new group of evaluators, by 
entering the e-mail addresses and other information of those evaluators to 
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the system (705). Each evaluator is assigned a subset (the "Response 
Subset Group") of the gathered suggestions (the "Response Population") in 
a random fashion. As samples are drawn from the Response Population, the 
suggestion number associated with each suggestion is used to cross- 
reference a response (or suggestion), a stimulus (or questionnaire) in other 
processes and operations in the system. 

In this embodiment of the invention, it is possible for the same sample 
(i.e., suggestion) to be assigned to different evaluators. The system only 
ensures that each evaluator has been assigned a Response Subset Group of 
equal size and that all suggestions in the Response Population have been 
distributed (707). The assignment of Response Subset Groups to the 
evaluators is recorded in the evaluator database 125. The system then 
transmits a Response Subset Group to each evaluator via e-mail. The e-mail 
account may either be on the system or on a third party e-mail system as 
determined by the evaluator when he registered with the system using the 
Open Account process 301 . In the cases where automated system 
evaluation process are used, such as statistical analysis, complex data 
correlation analysis, and knowledge extraction using machine learning tools, 
etc., the allocation of suggestions to various processes are made 
automatically; and the system synchronizes these various operations to 
optimize overall performance. 
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An example of the type of evaluation is to rate the suggestions: 
evaluators rank the suggestions contained in their respective Response 
Subset Groups from "best" to "worst" based on a preset numbering system, 
e.g., "best" equals "1" and "worst" equals 10. Evaluators submit their 
evaluation result to the system, which at a reasonable interval, gathers and 
integrates evaluation results from the group of evaluators. In the case of 
ranking data, a temporary array "Ranking Array" is used to organize the 
result, which eventually holds the consensus ranking for all evaluated 
suggestions. The Ranking Array is an associated array where the rate of 
each suggestion is linked to its suggestion number, e.g., referenced by a 
computer program as Ranking_Array[suggestion number][ranking]. 

In an embodiment where none of the suggestions is allowed to share 
the same ranking, the system processes all suggestions to determine their 
ranking by maintaining a running sum in the Ranking Array. This is 
accomplished by: a) retrieving the current ranking for each entry in the 
Ranking Array using the suggestion numbers; b) inserting the newly 
processed ranking to the ranking list retrieved in the previous step, one at a 
time; and c) placing the newly ranked entry into the enlarged Ranking Array, 
one at a time. The resulted Ranking Array contains a running total of all 
rankings of the evaluated suggestions, each identifiable by its suggestion 
number. 
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In another embodiment, a suggestion can have the same numeric 
ranking value as some other suggestions; such suggestions are deemed to be 
of equal value for the purpose of awarding any rewards. The reward monies 
may be distributed among the suggestioners who submitted the highest 
ranked suggestions. The evaluation results and the rewards data are 
subsequently used to update the evaluator database 1 25 and the 
suggestionee database 121. 

The system may choose to engage in additional rounds of evaluation 
(709); and it may also choose to run a number of different evaluation steps, 
such as statistical analysis, etc. outlined above. After completing all 
evaluation steps, the system queries the suggestionee database 121 to 
determine if the Commonality Flag was set TRUE by the suggestionee in the 
Create Suggestion-Stimuli Process 303 to allow sharing of the suggestions. 
If it was set TRUE, then the commonality database 131 is updated (71 1) and 
the evaluated suggestions are made available for viewing by other 
suggestionees who had also enabled their Commonality Flag for similar 
suggestion stimuli. 

(6) EVALUATED SUGGESTION OUTPUT TRAY 

The Evaluated Suggestion Output Tray 313 displays the evaluated 
suggestions output from the Suggestion Evaluation Process 307. The 
evaluated suggestions may be rank-ordered . Referring to Fig. 10, the 
Evaluated Suggestion Output Tray 313 involves a sequence of operations in 
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the preferred embodiment of the invention. At the host web site of the 
system, a suggestionee enters his user identification and password, and 
selects a stimuli (e.g., a questionnaire) for which he intends to view 
responses (1001). The suggestionee is then presented with a list of 
evaluated responses generated from the Suggestion Evaluation Process 307 
that are associated with the selected stimuli (1003). The suggestionee may 
select an individual suggestion, and request that additional information or 
clarification be provided by any of the various evaluators. This is 
accomplished by the system querying the evaluator database to determine 
which evaluators had enabled bi-directional communication with the 
suggestionee (1005). Once an evaluator or a group of evaluators are 
identified who had enabled such communication, the suggestionee may enter 
a clarification question (1007), which is then forwarded to the evaluator(s) 
via e-mail (1009). 

(7) NETWORK COMMONALITY DETERMINATION PROCESS 

The system enables a suggestionee to receive indirect suggestions to 
similar suggestion stimuli that are used by other suggestionees. This 
privilege of sharing suggestion feedback is only extended on a reciprocal 
basis to other suggestionees having set the Commonality Flag TRUE in the 
Create Suggestion-Stimuli Process 303, in the preferred embodiment of this 
invention. The system invokes the Network Commonality Determination 
Process 31 1 automatically at a preset reasonable interval. 
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Referring to Fig. 9, the Network Commonality Determination Process 
31 1 includes a sequence of steps in one embodiment of this invention. The 
system periodically and automatically parses the suggestionee database 121 
to determine if the Commonality Flag was set by new and existing 
suggestionees in the Create Suggestion-Stimuli Process 303 (901). Once 
the system detects a TRUE Commonality Flag, it copies the suggestion 
stimuli and the associated suggestion responses to the commonality 
database 131, from the Suggestion Stimuli Input Tray 305 and Raw 
Suggestion Output Tray 309, respectively (903). The system then again 
parses the suggestionee database 121, and compares all stimuli or 
questionnaires found in the suggestionee database 121 to those in the 
commonality database 131. If a match is found, then those suggestionees 
who had previously set the Commonality Flag TRUE are informed (e.g., via e- 
mail) of available indirect suggestions (905). These suggestions are then 
automatically copied to the respective Raw Suggestion Output Trays 309 of 
the participating suggestionees (907). 

(8) REDEEM AWARDS 

The Redeem Awards process 315 allows a suggestioner or evaluator 
to redeem their reward points which they earned by submitting a suggestion 
via the Suggestion Stimuli Input Tray 305, and by evaluating suggestions in 
the Suggestion Evaluation Process 307. A Reward Point may be defined to 
1/1 00 th of a US Dollar ($0.01), in an embodiment of this invention; in other 
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embodiments of the invention, reward points may be converted to goods or 
services at a fair market price. 

Referring to Fig. 1 1, the Redeem Awards process 315 includes a 
sequence of steps. A registered suggestioner or evaluator first enters his 
user identification and password at the host web site of the system and 
chooses Redeem Awards by clicking on a labeled button. The system then 
performs a query to the rewards database 127, and retrieves and displays 
the balance of the user's rewards account (1 101). The user then selects the 
amount of Reward Points to redeem (1 103). The Rewards Points can be 
redeemed to either cash or goods and services of equal value. If the user 
elects to convert Reward Points to cash, then the system prompts the user 
to enter the necessary information to process the transaction (e.g., bank 
routing numbers, credit card accounts, name, address, etc.) (1 105). The 
system then disburses the funds (e.g., prints a physical check and mails it 
via the postal system, or transfers funds electronically to a bank or credits a 
charge card account) (1 107). If the user elects to convert Reward Points to 
good or services, then the system prompts the user for the information 
necessary to complete the transaction. The system then updates the 
suggestioner database 123, the evaluator database 125, the suggestionee 
database 121, and the rewards database 127. In one embodiment, the 
system may post the amount of any award on the host web site. 
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Although only a few exemplary embodiments of this invention have 
been described in detail above, those skilled in the art will readily appreciate 
that many modifications are possible in the exemplary embodiments without 
materially departing from the novel teachings and advantages of this 
5 invention. Accordingly, all such modifications are intended to be included 
within the scope of this invention. 
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What is claimed is: 

1 . A system for collecting one or more suggestions from a first entity and 
providing access to collected suggestions to a second entity comprising a 
memory for storing one or more suggestions submitted by the first entity in 
response to a stimulus selected by the second entity and an interface 
permitting the second entity to access the stored suggestions. 

2. A method for obtaining suggestions, comprising selecting a stimulus 
to solicit suggestions from a first party, said suggestions being stored in a 
memory, and obtaining access to suggestions stored in the memory through 
an interface displayed by a server. 

3. A system of flow of suggestions, comprising: 

(a) at least one solicitor providing one or more stimuli for soliciting 
suggestions; 

(b) at least one respondent providing one or more raw suggestions in 
response to said one or more stimuli; and 

(c) a processor that is capable of organizing, evaluating, and otherwise 
processing said one or more raw suggestions, resulting in processed 
suggestions which are then delivered to said solicitor. 
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4. The system of claim 3, wherein said processed suggestions are 
substantially the same as said raw suggestions. 



5. The system of claim 3, wherein said processed suggestions are 
delivered to a second solicitor which provides one or more substantially 
similar stimuli. 

6. The system of claim 3, wherein said processed suggestions are 
delivered to a receiver which does not provide any stimuli and which is 
designated by said solicitor. 

7. The system of claim 3, further comprising a channel whereby said 
solicitor engage in direct communication with said respondent after receiving 
said processed suggestions. 

8. The system of claim 3, wherein said stimuli further comprises rewards 
incentives. 

9. The system of claim 3, wherein the flow of suggestions is via a 
computer network, and wherein a single interface on said network is 
provided to said each solicitor and to said each respondent to support said 
flow of suggestions. 
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10. The system of claim 3, further comprising a database having stored 
therein information of said solicitor, said respondent, said stimuli, and said 
raw and processed suggestions. 

1 1 . The system of claim 10, further comprising a user interface whereby 
said solicitor and said respondent access said information. 

12. A method for obtaining suggestions, comprising 

(a) providing one or more stimuli by at least one solicitor for soliciting 
suggestions; 

(b) gathering one or more raw suggestions from at least one 
respondent in response to said one or more stimuli; 

(c) organizing, evaluating, and otherwise processing said one or more 
raw suggestions, resulting in processed suggestions; and 

(d) delivering said processed suggestions to said solicitor. 

13. The method of claim 12, wherein said processed suggestions are 
substantially the same as said raw suggestions. 
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14. The method of claim 12, further comprising delivering said processed 
suggestions to a second solicitor which provides one or more substantially 
similar stimuli. 



15. The method of claim 12, further comprising delivering said processed 
suggestions to a receiver which does not provide any stimuli and which is 
designated by said solicitor. 

1 6. The method of claim 1 2, further comprising providing rewards 
incentives along with said stimuli. 

17. A method for providing suggestions, comprising 

(a) providing one or more stimuli by at least one solicitor for soliciting 
suggestions; 

(b) gathering one or more raw suggestions from at least one 
respondent in response to said one or more stimuli; 

(c) organizing, evaluating, and otherwise processing said one or more 
raw suggestions, resulting in processed suggestions; and 

(d) delivering said processed suggestions to a receiver designated by 
said solicitor. 

18. A system of flow of suggestions, comprising: 
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(a) at least one solicitor providing one or more stimuli for soliciting 
suggestions; 

(b) at least one respondent providing one or more raw suggestions in 
response to said one or more stimuli; and 

5 (c) a processor that is capable of organizing, evaluating, and otherwise 

processing said one or more raw suggestions, resulting in processed 
suggestions which are then delivered to a receiver designated by said 
solicitor. 
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